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Abstract 


This  paper  introduces  the  Knowledge  Value  Added/Real  Options  (KVA+RO) 
valuation  framework  for  use  in  information  technology  (IT)  portfolio  management 
within  the  Navy.  KVA+RO  is  designed  to  support  information  technology  (IT) 
portfolio  investment  decisions.  It  is  intended  to  empower  decision-makers  by 
providing  performance-based  data  and  analyses  like  the  Return  On  Investment 
(ROI)  on  individual  projects,  programs  and  processes  within  a  portfolio  of  IT 
investments.  Using  KVA  historical  data  as  a  platform,  potential  strategic 
investments  are  evaluated  with  real  options  analysis. 

The  first  section  discusses  limitations  of  existing  ROI  approaches.  The  paper 
then  presents  KVA+RO  methodology  and  framework,  reviews  core  concepts, 
underlying  assumptions,  metrics  and  potential  applications  to  the  IT  portfolio 
management  problem  in  the  DoD.  In  the  final  section,  the  KVA+RO  valuation 
framework  is  applied  to  Naval  Cryptologic  Carry  On  Program  (CCOP)  systems,  that 
are  used  in  the  Intelligence  Collection  Process  (ICP),  for  evaluation  of  signal 
intelligence  gathering  system  investments. 

Keywords:  Return  on  Investment  in  Information  Technology,  Real  Options, 
Business  Process  Reengineering,  Performance  Accounting,  ISR 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


-II 


Acknowledgements 


The  team  would  like  to  thank  the  ongoing  support  of  the  SPA  WAR  CCOPS 
team  headed  by  Wayne  Bratt  and  guided  by  the  leader  of  the  CCOPS  team  at  OP 
NAV  N20,  CDR  Joe  Sisson  and  LCDR  Brian  Prevo.  These  leaders  had  the  vision  to 
understand  the  dilemma  of  how  to  prioritize  CCOP  IT  investments  and  have  been 
enormously  supportive  of  our  efforts  to  provide  them  a  practical  and  robust 
methodology  to  do  so.  We  would  also  like  to  acknowledge  the  ongoing  support  of 
the  ultimate  users  of  this  methodology  who  operate  at  the  “point-of-the-spear”  such 
as  LCDR  Cesar  Rios  who  is  currently  operating  somewhere  in  the  war  zone  in  the 
middle  East  with  the  3'^'^  Expeditionary  Strike  Group. 

Finally,  we  would  like  to  thank  RADM  (retired)  Jim  Greene,  NPS  Acquisitions 
Chair,  for  his  ongoing  support  of  this  research  methodology  as  it  is  being  applied  in 
numerous  acquisition  applications.  His  direction  and  support  has  been  central  to  the 
success  of  all  our  research  efforts  using  this  methodology  in  the  Navy  context. 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


-IV' 


About  the  Authors 


LCDR  Cesar  Rios  is  a  Naval  Cryptologist  &  Information  Warfare  Officer.  His 
military  experience  includes  two  tours  on  the  staff  of  Commander  U.S.  Naval  Forces 
Central  Command,  Manama,  Bahrain,  where  he  served  as  the  Intelligence  Systems 
&  Special  Intelligence  Communications  Officer  and  Regional  Intelligence 
Engagement  Officer  (Bahrain,  Saudi  Arabia  and  Pakistan)  during  Operations 
ENDURING  FREEDOM  and  IRAQI  FREEDOM.  His  contact  information  is  classified 
and  he  is  currently  on  deployment  in  the  war  zone  in  the  Middle  East.  Dr.  Housel  will 
facilitate  contact  with  LCDR  Rios. 

Dr.  Tom  Housel  specializes  in  valuing  intellectual  capital  and  knowledge 
value  measurement.  He  is  currently  a  tenured  Full  Professor  for  the  Information 
Sciences  (Systems)  Department.  He  won  the  prestigious  Society  for  Information 
Management  award  for  best  paper  in  the  field  in  1986.  His  work  on  measuring  the 
value  of  intellectual  capital  has  been  featured  in  a  Fortune  cover  story  (1994, 
October  3)  and  Investor’s  Business  Daily,  numerous  books,  professional  periodicals, 
and  academic  journals  (most  recently  in  the  Journal  of  Intellectual  Capital  (Vol.  2), 
2005). 

Thomas  Housel 
Professor 

Graduate  School  of  Operational  and  Informational  Sciences 

Naval  Postgraduate  School 

Monterey,  CA  93943 

Tel;  (831)656-7657 

E-mail:  tihousel@nps.edu 

Dr.  Johnathan  Mun  is  the  founder  and  CEO  of  Real  Options  Valuation,  Inc., 
a  consulting,  training  and  software  development  firm  specializing  in  real  options, 
employee  stock  options,  financial  valuation,  simulation,  forecasting,  optimization  and 
risk  analysis  located  in  northern  California.  He  is  the  creator  of  the  Super  Lattice 
Solver  software.  Risk  Simulator  software  and  Employee  Stock  Options  Valuation 
software  at  the  firm.  He  has  also  authored  numerous  books,  including:  Real  Options 
analysis:  Tools  and  Techniques,  Real  Options  analysis  Course:  Business  Cases, 
Applied  Risk  Analysis:  Moving  beyond  Uncertainty,  and  others. 

Johnathan  Mun 
Research  Professor 

Graduate  School  of  Operational  and  Information  Sciences 

Naval  Postgraduate  School 

Monterey,  CA  93943 

Tel:  (925)271-4438 

E-mail:  icmun@nps.edu 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


-VI 


ACQUISITION  RESEARCH 
SPONSORED  REPORT  SERIES 


Integrated  Portfolio  Analysis:  Return  on  Investment  and  Real 
Options  Analysis  of  Intelligence  Information  Systems 
(Cryptologic  Carry  On  Program) 

30  September  2006 
by 

LCDR  Cesar  G.  Rios,  Jr.,  Naval  Postgraduate  School 
Dr.  Thomas  Housel,  Naval  Postgraduate  School 
Dr.  Johnathan  Mun,  Naval  Postgraduate  School 


Disclaimer:  The  views  represented  in  this  report  are  those  of  the  author  and  do  not  reflect  the  official  policy  position  of 
the  Navy,  the  Department  of  Defense,  or  the  Federal  Government. 


-VII- 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


Table  of  Contents 


I.  Introduction  1 

II.  Return  on  Investment  2 

III.  Beyond  ROI:  The  KVA+RO  Valuation  Framework  7 

A.  Overview  of  KVA+  RO  Framework  8 

B.  Knowledge  Value  Added  Methodology  9 

C.  Real  Options  Analysis  12 

D.  Potential  Applications  of  KVA  +  RO  Framework  14 

IV.  Proof-of-Concept:  USS  Readiness  Case  Study  16 

A.  Background  17 

B.  Applying  KVA  Methodology  19 

C.  KVA  Results  20 

D.  Real  Options  Analysis  23 

E.  Conclusions  and  Recommendations  26 

V.  Conclusion  28 

Appendix  1.  Calculation  Steps  in  KVA  Methodology  31 

Appendix  2.  Current  Limitations  of  KVA  Methodology  33 

Appendix  3.  Select  List  of  Organizations  Who  Have  Applied  KVA 
Methodology  35 

Appendix  4.  Case  Study  Handout  37 

Initial  Distribution  List  51 


-ix- 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


I.  Introduction 

The  United  States  has  experienced  dramatic  changes  in  national  security 
over  the  past  15  years,  shifting  away  from  the  conventional  threats  posed  during  the 
Cold  War  era  to  the  more  unconventional  threats  evidenced  by  the  tragic  events  of 
September  1 1  and  the  continuing  global  war  on  terrorism.  To  meet  the  challenges 
of  the  new  national  security  environment,  the  Department  of  Defense  (DoD)  plans  to 
spend  $1 .3  trillion  between  2005  and  2009  for  major  programs  ranging  from  new 
intelligence  programs  to  homeland  defense  and  military  operations  overseas.' 

With  lives  at  stake  and  billions  of  dollars  at  risk,  difficult  choices  are  made  by 
the  DoD  on  which  projects  to  fund  where  tradeoffs  occur.  Should  investments  be 
made  towards  personnel  or  investments  in  new  technology?  Should  more  funding 
be  allocated  towards  intelligence  collection  or  processing?  To  evaluate  and  select 
projects  returning  maximum  benefits,  new  measurement  tools  are  critical  to  properly 
define,  capture  and  measure  the  total  value  of  investments.  These  tools  must  be 
capable  of  capturing  data  across  a  spectrum  of  organizations  to  compare  processes, 
capabilities,  costs,  revenues  and  other  benefits.  In  addition,  they  must  incorporate 
elements  of  uncertainty  and  risk  that  are  inherent  in  predicting  the  future. 
Understanding  uncertainties  and  the  potential  impact  of  risks  can  significantly 
improve  the  likelihood  of  successful  investment  decisions.  A  team  at  the  Naval 
Postgraduate  School  (NPS)  applied  the  KVA+RO  valuation  framework  to  address 
those  issues.  . 


'  GAO,  Better  Support  of  Weapon  System  Program  Managers  Needed  to  Improve  Outcomes, 
November  2005,  p.  17. 
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II 


Return  on  Investment 


DoD  programs  often  experience  large  cost  increases  and  extended  delivery 
schedules,  resulting  in  estimates  of  time  and  money  being  inaccurate  by  20  to  50 
percent  or  higher.^  As  a  consequence,  the  ROI  on  DoD  IT  development  programs 
has  been  unpredictable  for  decades.  Statistics  for  the  private  sector  are  just  as 
disconcerting  (Figure  1),  with  large  technology  projects  having  a  dismal  success 
rate.  According  to  research  firm  The  Standish  Group,  IT  projects  with  budgets 
exceeding  $3  million  have  a  68  percent  chance  of  failing. 

Figure  1.  Rate  of  Successful  Project  Delivery 


(Source:  The  Standish  Group,  CHAOS  Survey,  Boston,  2004.) 


Fligh  failure  potential,  shrinking  revenues  and  smaller  budgets  have  forced 
organizations  to  spend  significant  resources  on  measuring  returns  from  the  millions 
of  dollars  invested  in  technology.  Various  approaches  have  been  adopted  to 
measure  ROI  at  corporate  and  sub-corporate  levels.  Corporate-level  approaches 


"  Ibid. 
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seek  to  determine  the  contribution  of  human  capital  and  IT  assets  on  the  overall 
performance  of  the  organization.  Sub-corporate-level  approaches  focus  internally 
on  the  core  processes  involved  in  the  production  of  organizational  output  and 
attempt  to  establish  a  measure  for  the  benefits  of  assets  within  each  process. 

Some  methodologies  are  traditional  financial  measures;  others  are  heuristical 
methods  supporting  quantitative  measures  using  subjective  estimates.  There  are 
also  probabilistic  approaches  using  statistical  and  mathematical  models  to  measure 
risk.  Although  methodologies  differ,  each  has  the  common  goal  of  providing 
managers  with  a  metric  of  value  for  IT  investments  as  summarized  in  Table  1 . 

Table  1.  Approaches  to  Measuring  Return  on  IT 


Level  of 
Analysis 

Approach 

Focus/ Assumptions 

Key  Advantages 

Limitations 

Aggregate 
Corporate 
(firm)  level 

Process  of 
Elimination 
(i.e.,  Knowledge 
Capital) 

•  Treats  effect  of  IT  on  ROI  as  a 
residual  after  accounting  for  other 
capital  Investments 

•  Uses  commonly  accepted  financial 
analysis  fechniques  and  existing 
accounting  data 

•  Cannot  drill  down  to  effects  of 
specific  IT  initiatives 

•  ROI  on  IT  difficult  to  measure 
directly 

Production  Theory 

•  Determines  IT  effects  through  Input/ 
output  analysis  using  regression 
modeling  techniques 

•  Economic  production  function  links 

IT  investment  input  to  productivity 
output 

•  Uses  econometric  analysis  on 
large  data  sets  to  show 
contributions  of  IT 

•  "Black-box"  approach  with  no 
intermediate  mapping  of  IT's 
contributions  to  outputs 

Resource  Based 

View 

•  Links  firm’s  core  capabiiities  wifh 
competifiveness 

•  Uniqueness  of  IT  resource  = 
competifive  advantage 

•  Uses  strafegic  advantage 
approach  to  IT  impacts 

•  Causal  mapping  between  IT 
investment  and  firm  competitive 
advantage  difficult  to  establish 

Corporate/ 

sub-corporate 

Option  Pricing  Model 

•  Determines  best  point  to  exercise  an 
option  to  invest  in  IT 

•  Timing  exercise  option  =  value 

•  Predicts  future  value  of  IT 
investment 

•  No  surrogate  for  revenue  at  sub¬ 
corporate  level 

Sub-corporate 
(Process)  level 

Family  of  Measures 
(i.e.,  Balanced 
Scorecard) 

•  Measures  multiple  indicators  to 
derive  unique  contributions  of  IT 

•  Captures  complexity  of  corporate 
performance 

•  No  common  unit  of  analysis/ 
theoretical  framework 

•  Multiple  indicators  required  to 
measure  performance 

Cost-Based 

(i.e.,  Activity  Based 

Costing) 

•  Uses  cosf  to  determine  value  of  IT 

•  Derivations  of  cost  =  value 

•  Captures  accurate  cost  of  IT 

•  No  surrogate  for  revenue  at  this 
level;  no  ratio  analysis 

Knowledge  Value 

Added 

(i.e.,  KVA) 

•  Allocates  revenue  to  IT 
proportionate  to  contributions  to 
process  outputs 

•  IT  contributions  to  output  ~  IT  value- 
added 

•  Allocates  revenue  and  cost  of  IT 
allowing  ratio  analysis  of  IT  value- 
added 

•  Not  directly  applicable  to  highly 
creative  processes 

Source:  Paul  Pavlou,  Thomas  Housel,  Waymond  Rodgers  and  Eric  Jansen.  “Measuring  the  Return  on 
Information  Technology:  A  Knowledge-based  Approach  for  Revenue  Allocation  at  the  Process  and  Firm 
Level,”  Journal  of  the  Association  for  Information  Systems  6,  no.  7  (July  2005):  199-226. 


Most  ROI  measures  focus  solely  on  financial  returns  at  the  corporate  level 
and  fail  to  capture  complete  benefit  streams  produced  within  organizational 
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processes  and  the  productive  assets  therein.  Limitations  with  current  attempts  at 
ROI  metrics  include: 

•  Capturing  IT-specific  benefits.  With  the  first  wave  of  new 
technology,  there  was  a  clear  distinction  of  benefits  received  from  that 
specific  technology.  Inputs  and  outputs  were  clearly  defined,  but 
returns  resulting  from  investments  in  mainframes  to  personal 
computers  to  client/server  to  enterprise  resource  planning  suite  to  the 
Web  were  indistinguishable  because  there  was  no  defensible  way  to 
allocate  revenue  to  these  IT  assets.  People,  processes  and  technology 
have  now  become  seamlessly  integrated — further  complicating  efforts 
to  develop  ROI  estimates  for  human  capital  assets  and  information 
technology  assets. 

•  Quantifying  intangible  human  capital  assets  (i.e.,  training,  skills, 
knowledge).  The  average  large  company  spends  the  equivalent  of 
two  percent  on  its  total  payroll  on  training,  according  to  the 
management  consult  firm  of  Bain  &  Co.^  How  does  that  expenditure 
impact  the  final  output  in  terms  of  creating  shareholder  value?  How  do 
training,  knowledge,  skill  and  education  all  contribute  to  the  bottom-line 
of  an  output? 

•  Defining  metrics  for  the  public  sector.  Public  sector  organizations 
must  evaluate  investments  on  the  overall  “value”  received.  What  does 
value  translate  to  in  the  public  sector?  What  capabilities  deliver  the 
greatest  value?  How  can  success  be  measured  in  the  public  sector? 
What  is  the  resulting  value  to  the  public  for  each  dollar  invested?  How 
can  public  sector  value  be  calculated  and  quantified  in  common  units 
such  as  money? 

Measurement  tools  for  assessing  the  performance  of  public  funding 
investments,  along  with  a  common  view  of  the  benefits  received  are  critical  given 
increased  regulations  and  pressures  for  better  accountability  and  transparency.  For 
example,  DoD  Directive  8115.01  issued  October  2005  mandates  the  use  of 
performance  metrics  based  on  outputs,  with  ROI  analysis  required  for  all  current  and 
planned  IT  investments.  NPS  developed  the  KVA+RO  framework  to  estimate  the 


^  Buchanan,  Webster,  Human  Capital  Review,^  Summer  2004,  p.  1. 

<http://www.websterb.com/pdf/first  page/hcr  review  european  firstpage.pdf?PHPSESSID=7797d0Q5O86Q63 

c9645389Q6e8a06Qc> 
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ROI  on  IT  investments  with  an  analytical  tool  set  that  also  supports  strategic, 
performance-based  investment  decisions. 
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III.  Beyond  ROI;  The  KVA+RO  Valuation 
Framework 


The  KVA-+RO  valuation  framework  measures  operating  performance,  cost- 
effectiveness  and  return  on  investments.  The  framework  also  facilitates  regulatory 
compliance  and  applies  portfolio-management  techniques  to  value  programs,  taking 
into  account  uncertainty  and  risk  in  estimating  future  benefits.  The  framework  is 
useful  for  the  DoD,  not  just  private  sector  firms.  Its  focus  on  the  outputs  of  core 
processes,  sub-processes  provides  several  benefits: 

•  Quantifies  value  of  specific  processes,  functions,  departments, 
divisions,  or  organizations  in  common  units  of  output. 

•  Provides  historical  data  on  costs  and  revenues  of  specific 
processes  and  specific  programs  within  organizations. 

•  Provides  a  methodology  that  will  facilitate  regulatory  compliance 
in  the  public  sector  with  legislation  (such  as  the  Clinger-Cohen 
Act  of  1996)  mandating  portfolio  management  for  all  federal 
agencies.  In  the  private  sector,  it  can  facilitate  compliance  with 
Sarbanes-Oxley  by  making  performance  among  corporate 
entities  more  transparent. 

•  Highlights  operational  efficiencies/inefficiencies  at  any  level  of 
analysis,  down  to  individual  employees  and  IT  systems. 

•  Leverages  current  and  future  portfolio  investments  by  estimating 
the  potential  total  value  created. 

Organizations  can  drill  down  to  understand  specific  processes  in  terms  of  the 
cost  of  each  process  and  its  contribution  to  the  bottom  line  with  the  KVA+RO 
framework.  Government  entities  can  use  the  framework  to  enhance  existing 
performance  tools.  On  the  corporate  side,  the  framework  can  be  used  to  value  any 
level  of  the  organization  from  specific  divisions  or  operating  units  down  to  individual 
employees  and  systems  to  determine  profitability. 
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A.  Overview  of  KVA+  RO  Framework 

The  KVA+RO  methodology,  including  the  general  data  collection  process,  is 
summarized  in  Table  2  (see  also  Appendix  1). 

Table  2.  NFS  Valuation  Framework 


DATA  COLLECTION 

KVA  METHODOLOGY 

REAL  OPTIONS  THEORY 

•  Collect  baseline 
data 

Step  1 :  Calculate  Time  to  Learn. 

Step  2:  Calculate  Value  of  Output  (K)  for  each  sub- 

Step  1 :  Risk  Identification 

List  of  projects  and  strategies  to  evaluate. 

•  Identify  sub¬ 
processes 

process. 

Step  3:  Calculate  Total  K  for  process. 

Step  2:  Risk  Prediction 

Base  case  projections  for  each  project. 

•  Research  market 
comparable  data 

•  Conduct  market 
analysis 

step  4:  Derive  Proxy  Revenue  Stream. 

Step  5:  Develop  the  Value  Equation  Numerator  by 

assigning  revenue  streams  to  sub-processes. 

Step  6:  Develop  value  equation  denominator  by 
assigning  costs  to  sub-processes. 

Step  3:  Risk  Modeiing 

Develop  static  financial  models  with  KVA  data. 

Step  4:  Risk  Analysis 

Dynamic  Monte  Carlo  simulation. 

•  Determine  key 
metrics 

Step  7,  8,  9:  Calculate  metrics: 

Return  on  Investment  (ROI) 

Return  on  Knowledge  Assets  (ROKA) 

Step  5:  Risk  Mitigation 

Framing  real  options. 

Return  on  Knowledge  Investment  (RKOI) 

Step  6:  Risk  Hedging 

Options  analytics,  simulation  &  optimization. 

Step  7:  Risk  Diversification 

Portfolio  optimization  and  asset  allocation. 

Step  8:  Risk  Management 

Reports  presentation  and  update  analysis. 

The  first  step  under  the  framework  is  data  collection  on  processes  and  sub¬ 
processes  required  to  produce  an  output.  Once  all  process  data  are  accurately 
documented,  they  are  supplemented  by  market  research  to  compare  cost  and 
revenue  data  to  establish  baseline  information.  The  KVA  methodology  is  then 
applied  to  estimate  value  and  costs  for  each  process.  Cost-per-unit  of  output 
calculated  by  KVA,  in  conjunction  with  price-per-unit  of  output  estimates,  provides 
raw  data  required  for  ROI  analysis.  In  the  final  step  of  the  framework,  real  options 
analysis  is  conducted  to  estimate  the  future  value  and  risks  of  potential  investments. 
Alternative  scenarios  are  run,  enabling  decision-makers  to  assess  risk,  leverage 
uncertainty  and  limit  downside  risk.  Principles  of  KVA  and  RO  are  discussed  further 
in  the  next  sections. 
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B.  Knowledge  Value  Added  Methodology 

A  new  paradigm  in  sub-corporate  performance  analytics,  KVA  measures  the 
value  provided  by  human  capital  assets  and  IT  assets  by  analyzing  an  organization, 
process  or  function  at  the  process  level.  It  provides  insights  into  each  dollar  of  IT 
investment  by  monetizing  the  outputs  of  all  assets,  including  intangible  knowledge 
assets.  By  capturing  the  value  of  knowledge  embedded  in  an  organization’s  core 
processes,  employees  and  IT,  KVA  identifies  the  actual  cost  and  revenue  of  a 
product  or  service.  Because  KVA  identifies  every  process  required  to  produce  an 
output  and  the  historical  costs  of  those  processes,  unit  costs  and  unit  prices  of 
products  and  services  are  calculated.  An  output  is  defined  as  the  end  result  of  an 
organization’s  operations;  it  can  be  a  product  or  service  as  shown  in  Figure  2  below. 

Figure  2.  Measuring  Output 


KVA  has  been  applied  in  over  100  organizations  in  the  public  and  private 
sectors,  ranging  in  size  from  under  20  employees  to  thousands,  for  the  past  14 
years.  The  methodology  has  been  applied  in  35  areas  within  the  DoD,  from  flight 
scheduling  applications  to  ship  maintenance  and  modernization  processes. 

As  a  performance  tool,  the  methodology: 
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•  Compares  all  processes  in  terms  of  relative  productivity 

.  Allocates  revenues  to  common  units  of  output 

.  Measures  value  added  by  IT  by  the  outputs  it  produces 

.  Relates  outputs  to  cost  of  producing  those  outputs  in  common  units 

.  Provides  common  unit  measures  for  organizational  productivity 

Based  on  the  tenets  of  complexity  theory,  KVA  assumes  that  humans  and 
technology  in  organizations  add  value  by  taking  inputs  and  changing  them  into 
outputs  through  core  processes.'*  The  amount  of  change  an  asset  or  process 
produces  can  be  a  measure  of  value  or  benefit.  Additional  assumptions  include: 

.  Describing  all  process  outputs  in  common  units  (i.e.,  the  time  it  takes 
to  learn  to  produce  the  required  outputs)  allows  historical  revenue  and 
cost  data  to  be  assigned  to  those  processes  at  any  given  point  in  time. 

.  All  outputs  can  be  described  in  terms  of  the  time  required  to  learn  how 
to  produce  them. 

.  Learning  Time,  a  surrogate  for  the  knowledge  required  to  produce 

process  outputs,  is  measured  in  common  units  of  time.  Consequently, 
Units  of  Learning  Time  =  Common  Units  of  Output  (K). 

.  Common  unit  of  output  makes  it  possible  to  compare  all  outputs  in 
terms  of  cost-per-unit  as  well  as  price-per-unit,  because  revenue  can 
now  be  assigned  at  the  sub-organizational  level. 

.  Once  cost  and  revenue  streams  have  been  assigned  to  sub- 

organizational  outputs,  normal  accounting  and  financial  performance 
and  profitability  metrics  can  be  applied. 

Describing  processes  in  common  units  also  permits  market  comparable  data 
to  be  generated,  particularly  important  for  non-profit  organizations  such  as  the  DoD. 
Market  comparable  data  from  the  commercial  sector  can  be  used  to  estimate  price 
per  common  unit,  allowing  for  revenue  estimates  of  process  outputs  for  non-profits. 
This  also  provides  a  common-units  basis  to  define  benefit  streams  regardless  of 
process  analyzed. 
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KVA  differs  from  other  ROI  models  because  it  allows  for  revenue  estimates — 
enabling  use  of  traditional  accounting,  financial  performance  and  profitability 
measures,  and  prospective  financial  methods  such  as  real  options  analysis.  Figure  3 
demonstrates  how  KV  A  compares  to  traditional  cost  accounting  by  providing  costs 
per  process  instead  of  cost  per  category.  Figure  4  provides  a  comparison  of 
traditional  corporate  level  revenue  information  while  KVA  provides  this  kind  of 
information  at  the  subcorporate  level  by  taking  the  corporate  level  revenue  and 
allocating  it  to  subcorporate  process  outputs. 

Figure  3.  Comparison  of  Traditionai  Accounting  versus  Process  Based 

Costing 


I 

Sr 


I 


1 


Traditional  Accounting 

KVA  Process  Costing 

Compensation 

$5,000 

Review  Task 

$1,000 

Benefits/OT 

1,000 

Determine  Op 

1,000 

Supplies/Materials 

2,000 

input  Search  Function 

2,500 

Rent/Leases 

1,000 

Search/Coliection 

1,000 

Depreciation 

1,500 

Target  Data  Acq 

1,000 

Admin.  And  Other 

900 

Target  Data  Processing 

2,000 

Total 

$11,400 

Format  Report 

600 

Quality  Control  Report 

700 

Transmit  Report 

1,600 

Total 

$11,400 

A 


J 


''  Thomas  Housel  and  Art  Bell,  Measuring  and  Managing  Knowiedge  (Boston:  McGraw-Hill,  2001), 
92-93. 
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Figure  4.  Comparison  of  Outputs  Traditional  Accounting  Benefits  (Revenues) 

versus  Process  Based  Value 


Traditional  Accounting/ 
Finance  Measure 

KVA  Process  Value  Measure 

Sales/Revenues 

Common  units  of  output 

Product  price 

Market  comparables:  Price-per-unit  of  output 

Total  Revenues 

Total  units  of  output  X  price-per-unit  =  total 
revenue  surrogate 

KVA  can  rank  processes  in  terms  of  the  degree  to  which  they  add  value  to  the 
organization  or  its  processes.  This  assists  decision-makers  to  identify  what 
processes  are  value-added — those  that  will  most  likely  accomplish  a  mission,  deliver 
a  service,  or  meet  customer  demand.  Value  is  quantified  in  four  key  metrics:  Return 
on  Knowledge  (ROK),  Return  on  Knowledge  Assets  (ROKA),  Return  on  Knowledge 
Investment  (ROKI)  and  Return  on  Investment  (ROI). 

C.  Real  Options  Analysis 

Real  options  analysis  incorporates  strategic  planning  and  analysis,  risk 
assessment  and  management,  and  investment  analysis.  As  a  financial  valuation 
tool,  real  options  allows  organizations  to  adapt  decisions  to  respond  to  unexpected 
environmental  or  market  developments.  As  a  strategic  management  tool,  real- 
options  is  a  strategic  investment  valuation  tool  affording  decision-makers  the  ability 
to  leverage  uncertainty  and  limit  risk.  Real-options  can  be  used  to: 
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Identify  different  corporate  investment  decision  pathways  or  projects 
that  management  can  consider  in  highly  uncertain  business  conditions; 


.  Value  the  feasibility  and  financial  viability  of  each  strategic  decision 
pathway; 

.  Prioritize  pathways  or  projects  based  on  qualitative  and  quantitative 
metrics; 

•  Optimize  strategic  investment  decisions  by  elevating  different  decision 
paths  under  certain  conditions  or  determine  how  a  different  sequence 
of  pathways  can  lead  to  the  optimal  strategy; 

.  Time  effective  execution  of  investments  and  find  the  optimal  trigger 
values  and  cost  or  revenue  drivers;  and 

Manage  existing  or  develop  new  options  and  strategic  decision 
pathways  for  future  opportunities.® 

Different  kinds  of  options  are  used  in  a  variety  of  ways  across  a  number  of 
industries  as  portrayed  in  Figure  5. 


Figure  5.  Types  of  Real  Options  and  Industry  Applications 


Types  of  Options 

Industry  Applications/Users 

•  Option  to  Wait 

•  DoD/Acquisitions,  Force  Mix 

(Proof  of  concept,  right  of  first  refusal,  getting  more  info) 

•  COOPS  Portfolio  Analysis 

•  Option  to  Execute 

•  Aeronautics/Boeing,  Airbus 

(Contracts  in  place  which  may/not  be  executed) 

•  Oil  and  Gas/BP,  Shell 

•  Abandonment  Option 

•  High  Tech/Intel 

(exit  and  salvage) 

•  Pharmacology/Merck,  Pfizer 

•  Expansion  Option 

•  R&D  Portfolios/Motorola,  Unilever 

(platform  technologies,  acquisitions,  open  architecture) 

•  IT  Infrastructure/Credit  Suisse 

•  Contraction  Option 

•  Electricity/Peaker-Plants 

(outsourcing,  alliances) 

•  Acquisitions/Seagate 

•  Compound  Option  (platform  options) 

•  Contracts/Syngenta,  GM 

•  Sequential  Options  (stage-gate  development,  R&D,  phased 

options) 

Although  there  are  many  approaches,  the  methodology  used  in  the  KVA+RO 
valuation  framework  is  one  developed  by  leading  expert  Dr.  Johnathan  Mun.  Dr. 
Mun’s  real-options  approach  consists  of  eight  steps  visually  represented  in  Figure  6. 
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Figure  6.  Real  Options  Analysis  Steps 


1.  List  of  projects  and 
strategies  to  evaluate 


Start  with  a  list  of  projects  or 
strategies  to  be  evaluated. . .  these 
projects  have  already  been 
through  qualitative  screening 


2.  Base  case  projections  for  each 
project 


Historical  Data  Gathering 
Time  Series  Forecasting 


...will  the  assistance  of  time- 
series  forecasting  and 
historical  data. . . 


3.  Develop  static  financial 
models  with  ICVA  data 


. .  .the  user  generates  a  traditional 
series  of  static  base  case  financial 
(discounted  cash  flow)  models  for 
each  project. . . 


4.  Dynamic  Monte  Carlo 
simulation 


. .  .sensitivity  and  scenario  analysis 
coupled  with  Monte  Carlo  simulation  is 
added  to  the  analysis  and  the  financial 
model  outputs  become  inputs  into  the  real 
options  analysis... 


5.  Framing  Real  Options 


. .  .the  relevant  projects  are 
chosen  for  real  options 
analysis  and  the  project  or 
portfolio  real  options  are 
framed. . . 


6.  Options  analytics,  simulation 
and  optimization 


Simulation  Lattice 


. .  .real  options  analytics  are  calculated 
through  binomial  lattices  and  closed-form 
partial-differential  models  with 
simulation. . . 


7.  Portfolio  optimization  and  asset 
allocation 


. . .  stochastic  optimization  is  the  next 
optional  step  if  multiple  projects  exist  that 
requires  efficient  asset  allocation  given 
some  budgetary  constraints. . .  useful  for 
strategic  portfolio  management... 


8.  Reports  presentation  and 
update  analysis 


B. .  .create  reports,  make  decisions, 
and  do  it  all  again  iteratively  over 
time. . . 


D.  Potential  Applications  of  KVA  +  RO  Framework 

The  strategic  value  of  real  options  for  the  DoD  is  that  it  offers  alternative 
paths  to  decision-makers.  In  a  dynamic  and  uncertain  environment  where 
investment  decisions  must  be  flexible  and  fluid,  real-options  offers  insights  into 
alternative  paths  and  how  they  relate  to  unique  DoD  requirements.  A  tool  to 
augment  existing  performance  tools,  KVA+RO  can  be  applied  in  many  areas  in  the 
DoD  as  summarized  in  Table  3. 


^Jonathan  Mun,  “Using  Real-options  Software  to  Value  Complex  Options,”  Financial  Engineering 
Times  T1  (September/October  2002)  pages  23-26. 
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Table  3.  Potential  DoD  Applications  of  KVA  and  Real  Options 


Application 

Activity  Based 
Costing  (ABC) 
Enhancement 

•  KVA  provides  a  way  to  define  common  units  of  output  of  all  processes. 

•  RO/KVA  provides  a  way  to  compare  outputs  per  cost  value  flows  (not  just  costs). 

•  KVA  provides  a  cost-per-common-unit  of  output  across  all  processes. 

OMB  Circular  A- 
76  Comparisons 

•  RO/KVA  could  enhance  outsourcing  comparisons  between  the  Government’s  Most 
Efficient  Organization  (MEO)  and  private  sector  alternatives. 

JCIDS  and  DAS 

•  RO  and  RO/KVA  present  themselves  throughout  JCIDS  requirements  generation  and 
the  Defense  Acquisition  System  (e.g.,  DOTMLPF  vs.  New  Program/Service  solution, 

Joint  Integration,  Analysis  of  Material  Alternatives  (AMA),  Analysis  of  Alternatives  (AoA) 
and  Spiral  Development) 

CCOPS  Portfolio 
Analysis 

•  RO/KVA  theory  applies  to  Spiral  Development  and  various  organizational 

configurations,  as  well  as  the  Importance  of  how  Integrated  and  Open  Architectures 
become  RO  multipliers. 

Source:  Cesar  Rios,  Thomas  Housel  and  Johnathan  Mun,  “Real  Options  and  KVA  in  Military 
Strategy  at  the  United  States  Navy,”  in  Modeling  Risk:  Applying  Monte  Carlo  Simulation,  Real 
Options  Analysis,  Forecasting,  and  Optimization  Techniques,  ed.  Johnathan  Mun  (New  York: 
Wiley  Finance,  2006)  441-452. 
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IV 


Proof-of-Concept;  USS  Readiness  Case  Study 


The  valuation  framework  was  applied  to  the  fictitious  US  Navy  warship,  USS 
Readiness.  Our  case  study  focused  on  the  Cryptologic  Carry-On  Program  (COOP) 
portfolio  of  intelligence  information  systems  and,  in  particular,  the  ship-borne  signals 
intelligence  collection  process.  KVA+RO  allows  for  analysis  of  existing  and  future 
COOP  systems  on  intelligence,  search  and  rescue  (ISR)  activities,  processes  and 
operations  for  each  system  in  the  portfolio.  Individual  COOP  systems  in  the  portfolio 
can  be  compared  once  baseline  data  is  created,  enabling  decision-makers  to  make 
financial  decisions  and  projections  based  on  quantitative  data. 

A.  Background 

The  USS  Readiness  is  outfitted  to  conduct  Intelligence,  Surveillance,  and 
Reconnaissance  (ISR)  missions  and  has  a  contingent  of  information  warfare 
operators  performing  intelligence  collection  processes  utilizing  CCOP  systems. 
Principal  sub-processes  in  the  ICP  are  shown  in  the  Figure  7. 


-17- 


Figure  7.  The  Intelligence  Collection  Process 


P3 


The  warship  is  equipped  with  four  CCOP  systems:  A,  B,  C,  and  D.  CCOP 
systems  may  be  used  in  a  single  sub-process  or  across  sub-processes,  and  some 
systems  such  as  CCOP  A  are  highly  complex  with  multiple  subsystems  as  shown  in 
Table  4  below.  Each  sub-process  is  further  broken  down  into  individual  actions  that 
may  be  required  to  perform  the  sub-process  in  the  intelligence  collection  process. 
For  example,  sub-process  “Target  Data  Processing”  can  be  broken  down  into  a 
number  of  human-based  tasks  requiring  no  automation. 
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Table  4.  USS  READINESS  CCOP  Systems 


SUB-PROCESS  NAME 

CCOP  A 

CCOP  B 

CCOP  C 

CCOP  D 

P1 

Review  Request/Tasking 

X 

P2 

Determine  Op/Equip  Mix 

X 

P3 

Input  Search  Function/Coverage  Pian 

X 

P4 

Search/Coiiection  Process 

X 

X 

P5 

Target  Data  Acquisition/Capture 

X 

X 

P6 

Target  Data  Processing 

X 

X 

X 

X 

P7 

Target  Data  Anaiysis 

X 

X 

X 

P8 

Format  Data  for  Report  Generation 

X 

P9 

QC  Report 

X 

P10 

Transmit  Report 

X 

B.  Applying  KVA  Methodology 

KVA  methodology  was  applied  to  quantify  the  value  added  by  CCOP 
systems,  information  warfare/cryptologic  operators,  and  the  enabling  ship-borne 
system  infrastructure  with  which  they  interact.  Value  provided  by  human  capital 
elements  were  compared  to  IT  elements  to  measure  efficiency  (productivity)  and 
effectiveness  (profitability).  All  assets,  sub-processes,  and  outputs  are  first 
identified. 

•  Asset  analysis  encompasses  all  value  and  cost  data  related  to  each  asset 
in  the  process,  human  capital  or  IT  asset. 

•  Sub-process  analysis  includes  a  detailed  breakdown  of  the  ICP  to 
include  the  time-to-learn,  how  to  perform  each  sub-process,  and  number 
of  executions  for  each  sub-process. 

•  Process  outputs  are  established  via  time-to-learn  estimates,  including  the 
total  number  of  aggregated  process  outputs  and  a  surrogate  revenue 
stream  used  to  monetize  the  outputs. 

Asset  values  and  costs  are  then  allocated  throughout  the  sub-processes  in 
which  they  contribute  to  the  production  of  outputs.  The  time-to-learn  (knowledge 
embedded  in  each  sub-process)  is  multiplied  by  the  number  of  executions  of  that 
sub-process,  and  the  figure  serves  as  a  basis  for  revenue  allocation  at  the  sub¬ 
process  level.  Costs  are  calculated  by  multiplying  the  time  it  takes  to  produce  the 
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process  output  times  the  salary  of  those  producing  it  and  the  cost  per  usage  of  the 
IT  asset.  Costing  typically  does  not  include  the  cost  of  fixed  assets  as  these  costs 
are  typically  used  as  a  constant  weighting  factor.  Therefore,  these  costs  usually  do 
not  affect  the  relative  performance  estimates  for  the  various  sub-processes. 
Performance  ratios  such  as  ROKA  and  ROKI  can  be  calculated  after  costs  and 
benefits  for  each  sub-process  are  defined. 

C.  KVA  Results 

KVA  analysis  was  used  to  compare  two  example  sub-processes:  “Search  and 
Collect”  (P4)  and  “Format  Data  for  Report  Generation”  (P8).  Results  are 
summarized  in  Tables  5  and  6;  issues  were  identified  at  the  portfolio,  program  and 
process  levels.  The  data  raises  questions,  as,  noted  for  each  process. 
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1.  Program  Level  Issues 


Table  5.  USS  READINESS  Summary  KVA  Results 
_ Return  on  Knowledge  (ROK)  _ 


Sub-Process 

CCOP  A 

CCOP  B 

CCOP  c 

CCOP  D 

ROK 

Review  Request/Tasking 

P1 

168.54% 

168.54% 

Determine  Op/Equip  Mix 

P2 

166.86% 

166.86% 

input  Search  Function/Coverage  Pian 

P3 

152.91% 

152.91% 

Search/Coiiection  Process 

P4 

930.03% 

148.15% 

590.13% 

Target  Data  Acquisition/Capture 

P5 

290.15% 

147.71% 

228.23% 

Target  Data  Processing 

P6 

319.39% 

162.59% 

436.13% 

28.18% 

142.41% 

Target  Data  Anaiysis 

P7 

149.98% 

534.76% 

34.55% 

121.42% 

Format  Data  for  Report  Generation 

P8 

143.34% 

143.34% 

QC  Report 

P9 

315.88% 

315.88% 

Transmit  Report 

P10 

148.75% 

148.75% 

ROK  for  Total  Process 

278.59% 

152.81% 

485.44% 

31.37% 

196.27% 

•  CCOP  D  is  a  cost-heavy  system  that  executes  very  few  times  with 
negative  ROKs  throughout  the  sample  period,  as  seen  in  Table  5. 

■  Is  CCOP  D  appropriate  for  this  platform  and  mission? 

■  What  is  a  less  expensive  alternative  to  CCOP  D? 

■  Are  all  operators  appropriately  trained  in  the  use  of  CCOP  D? 
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2.  Process  Level  Issues 


Table  6.  USS  READINESS  Summary  KVA  Results 


Return  on 

<now 

edge  Investment  (ROKI) 

Sub-Process 

CCOP  A 

CCOP  B 

CCOP  c 

CCOP  D 

ROKI  (ROI) 

Review  Request/Tasking 

pi 

68.54 

22.11 

Determine  Op/Equip  Mix 

P2 

66.86 

20.89 

input  Search  Function/Coverage  Pian 

P3 

52.91 

-18.44 

Search/Coiiection  Process 

P4 

830.03 

48.15 

239.01 

Target  Data  Acquisition/Capture 

P5 

190.15 

47.71 

47.28 

Target  Data  Processing 

P6 

219.39 

62.59 

336.13 

-71.82 

36.67 

Target  Data  Anaiysis 

P7 

49.98 

434.76 

-65.45 

21.25 

Format  Data  for  Report  Generation 

P8 

43.34 

-20.37 

QC  Report 

P9 

215.88 

79.19 

Transmit  Report 

P10 

48.75 

-17.37 

Metrics  for  Aggregated 

178.59 

52.81 

385.44 

68.63 

109.9 

•  The  Search  and  Collect  process  (P4)  is  knowledge-intensive,  requiring 
IT  and  human  capital  asset  investments  to  complete  each  sub-process 
in  Table  6.  Moreover,  each  sub-process  output  necessitates  many 
executions  of  the  sub-process. 

■  Could  an  even  higher  return  be  achieved  with  further  automated 
search  and  collection  systems  or  more  operators? 

■  Should  the  amount  of  knowledge  in  humans  and  IT  be 
adjusted? 

■  Could  a  broader  range  of  training  allow  operators  to  perform 
more  functions? 


•  The  Search  and  Collect  process  (P4)  is  a  high  performer,  with  an 
overall  return  of  239%  compared  to  a  -20.37%  return  for  the  Format 
Data  for  Report  Generation  process  (P  8). 
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What  accounts  for  the  discrepancy  in  the  returns  received  on 
each  process? 


•  The  Format  Data  for  Report  Generation  process  (P  8)  only  executes 
once  per  intelligence  report  (process  output)  with  nearly  one-third  of  all 
operators  assigned  to  this  sub-process  one-fifth  of  the  total  human 
cost. 

■  What  causes  this  low  efficiency  level? 

•  The  Format  Data  for  Report  Generation  process  (P  8)  is  more 
automated  than  P4. 

■  Could  this  process  be  further  automated  or  performed  by  other 
operators  to  yield  higher  efficiency  and  effectiveness  levels? 

Answers  to  these  questions  could  help  program  managers  allocate  funds  to 
new  systems  or  to  existing  systems  to  improve  CCOP  products  or  to  eliminate  a 
system  from  the  CCOP  portfolio.  Results  could  also  be  used  to  tailor  manning  and 
training  requirements  of  ISR  crews  deploying  CCOP  systems.®  (See  Appendix  2  for 
possible  limitations  of  this  KVA  analysis) 


D.  Real  Options  Analysis 

Real  options  analysis  was  performed  to  determine  the  prospective  value  of 
three  basic  options  (summarized  in  Table  7  below)  over  a  three-year  period  using 
KVA  data  as  input  for  the  analysts.  Three  potential  scenarios  were  identified. 


^  This  case  study  revealed  a  few  limitations  to  implementation  of  KVA  to  the  Intelligence  Collection 
Process  as  modeled  in  for  the  USS  READINESS.  Please  see  Appendix  2  of  the  complete  case 
study  for  limitations  and  issues  being  addressed. 
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Table  7.  CCOP  Strategic  Scenarios 


Option  A 

Remote  to  Shore 

Option  B 

Direct  Support 

Option  C 

Permanent  SSES 

•  Data  viewed  from  geographically  remote  center. 

•  Intelligence  collection  processing  from 
consolidated  center  requires  less  intelligence 
personnel  on  ships. 

•  Consolidating  capabilities  into  central  center 
popular  movement  to  cut  costs  and  provide  more 
shore  based  operations  to  support  war-fighting 
capabilities. 

•  Similar  to  consolidation  of  service  operations  in 
businesses  into  larger  and  fewer  call  centers. 

•  CCOP  equipment  &  operators 
move  from  ship  to  ship 
whenever  a  ship  came  into  port 
for  maintenance,  repair  or 
modernization. 

•  Fewer  sets  of  CCOP  equipment 
and  operators  required  to 
service  intelligence  gathering 
needs  of  the  fleet. 

•  CCOP  systems  and  operators 
assigned  to  given  ships  at  all 
times. 

•  Requires  more  operators  and 
COOP  systems. 

•  Potential  costs  increases, 
provides  more  control  of 
intelligence  capability  by  the 
ships  and  fleet  commanders. 

Each  strategic  scenario  is  explored  further,  below  in  Figure  8. 
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Figure  8.  Real  Options  Analysis  of  Strategic  Scenarios 


•  Remove  all  but  3  staff  placing  3  ops  in 
Fleet  Remote  Operations  Centers 
(FROCs)  on  shore. 

•  Remove  all  CCOP  adding  CCOP  E, 
which  would  enable  remote  operations. 

•  Centers  would  operate  year  round.  2 
ships  remoted. 


Direat  Support 


•  All  Equipment  and  Operators  are  modular  & 
mobile. 

•  Each  team  &  equipment  can  be  redeployed 
after  each  6-month  tour. 

•  Some  ships  will  need  to  be  “pre-groomed  to 
accept  DIRSUP  architecture. 

•  13  ships  currently  DIRSUP  enabled. 


facilities  to 
house  FROCs 
and  remote  ops 
for  2  ships. 


•  Add  2  more  ships 
to  the  DIRSUP 
inventory. 


Expand  FROCs 


’  Stop  after  Phase  I 


»  Expand  by  5  ships. 


•  Stop  after  Phase  I 


Expand  FROCs; 
add  remainder  DDGs 
(14)  for  28  total  ships. 


to  house  more 

*  Phase  I 

•  Use  existing 

operators,  add  10  1— ^ 

more  ships 

Exit 

Stop  after  Phase  II 


•  Add  remainder  8 
ships  to  DIRSUP 
program. 


Stop  after  Phase  II 


•  Expand  by 
remaining  6  ships. 


Permanent  SSES 


'  All  operators  &  equipment  permanently 
assigned  to  ship  whether  deployed  or  in  port. 

'  Permanent  teams  include  3  extra  operators  and 
one  extra  management  team  member. 

'  15  ships  currently  equipped  with  Perm.  SSES. 


•  Add  2  more  ships 
to  permanent 
SSES  inventory. 


»  Expand  by  5  ships 


’  Stop  after  Phase  I 


•  Stop  after  Phase  II 


Results  of  the  real  options  analysis  in  Table  8  below  indicate  that  Option  C 
delivers  the  highest  value  at  $15.2  million.  Although  apriori,  Options  A  and  B  were 
expected  to  have  significant  cost  savings,  it  is  possible  to  see  greater  total  value 
(with  much  lower  volatility,  or  risk),  for  Option  C  with  RO  analysis.  Fleet  and  Ship 
Commanders  who  intuitively  preferred  Option  C  because  it  permitted  greater  control 
of  intelligence  assets  for  specific  operations  now  have  objective  data  to  help  them 
review  their  preferred  option.  This  is  not  to  say  that  the  other  options  might  provide 
greater  strategic  value  in  the  long  run  once  they  are  implemented  with  more 
productive  CCOPs  assets  and  lower  volatility  based  on  overcoming  the  initial 
decrements  in  the  learning  curve  of  a  new  process  implementation.  (See  Appendix  3 
for  a  list  of  companies,  organizations  using  the  real  options  analysis  technique.) 
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Table  8.  Summary,  Real  Options  Analysis  Results 


Option  A 

Option  B 

Option  C 

PV  Option  Cost  (Year  1 ) 

$348,533 

$1,595,697 

$1,613,029 

PV  Option  Cost  (Year  2) 

$4,224,487 

$3,043,358 

$4,494,950 

PV  Option  Cost  (Year  3) 

$3,688,994 

$10,105,987 

$8,806,643 

PV  Revenues 

$24,416,017 

$33,909,554 

$38,820,096 

PV  Operating  Costs 

$16,220,188 

$16,765,513 

$9,951,833 

PV  Net  Benefit 

$8,195,829 

$17,144,041 

$28,868,264 

PV  Cost  to  Purchase  Option 

$425,000 

$169,426 

$72,611 

Maturity  Years 

3.00 

3.00 

3.00 

Average  Risk-Free  Rate 

3.54% 

3.54% 

3.54% 

Dividend  Opportunity  Cost 

0.00% 

0.00% 

0.00% 

Voiatiiity 

26.49% 

29.44% 

15.04% 

Total  Strategic  Value  with  Options 

$1,386,355 

$4,466,540 

$15,231,813 

E.  Conclusions  and  Recommendations 

Applying  the  KVA+RO  framework  to  the  USS  READINESS  demonstrates  how 
defensible  and  relatively  objective  metrics  could  be  derived  for  analysis  of  each 
CCOP’s  ROI  performance  in  the  portfolio/  (See  Appendix  4  that  provides  a 
teaching  case  study  based  on  this  research.) 

Based  on  the  results  of  our  initial  research,  we  make  several 
recommendations: 

•  Expand  scope  of  initial  study.  KVA  methodology  should  be 
applied  and  analyzed  over  a  larger  sampling  period  to  accurately 
measure  the  impact  of  CCOP  systems.  A  larger  study  should  be 
conducted  on  CCOP  systems  at  the  Carrier  Strike  Group  (CSG)  or 
Expeditionary  Strike  Group  (ESG)  level  over  the  course  of  one 
deployment  to  begin  establishing  performance  baselines  for 
systems  and  processes.® 

•  Collect  additional  process  data.  Supplemental  data  on  human 
and  automated  processes  should  be  collected  to  attain  near  real- 


^  KVA  analysis  was  conducted  on  a  limited  set  of  data.  To  obtain  a  more  comprehensive  picture  of 
CCOP  system  contribution,  multiple  iterations  of  this  analysis  would  have  to  be  run  across  the 
Navy-wide  enterprise  of  intelligence  collection  platforms  to  obtain  a  comprehensive  understanding 
of  CCOP  program  contribution. 

®  Currently  in  process  with  the  Third  Expeditionary  Strike  Group. 
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time  performance  data  reporting.  Automated  logging  of  system 
utilization  and  performance  are  readily  available  in  many  business 
applications.  Adapting  such  mechanisms  for  use  with  CCOP 
systems  would  facilitate  the  performance  analysis. 

•  Implement  KVA  software  for  real-time  analysis.  Although 
several  accounting  software  packages  have  included  KVA 
analytical  capabilities,  the  NPS  research  team  has  identified 
GaussSoft  KVA  software  as  the  most  comprehensive  software 
platform  for  conducting  the  level  of  analysis  required  by  DoD 
program  managers.  Implementing  GaussSoft  software  allows 
real-time  system  and  process  inputs  to  be  received,  as  well  as 
proof-of-concept  and  a  testing  of  the  operational  capabilities  of  the 
software  to  be  performed.  ^ 

•  Expand  research  study  to  Include  other  public-  and  private- 
sector  organizations.  An  extensive  research  study  should  be 
conducted  on  the  Market  Comparables  Approach  to  include  a 
valuation  study  of  the  intelligence  products  produced  by  private 
military  corporations,  along  with  competitive  and  business 
intelligence  organizations,  to  achieve  a  baseline  price-per-unit  of 
output  metric.  One  of  the  study’s  primary  objectives  would  be  to 
develop  universally  accepted  descriptions  of  embedded 
knowledge  and  required  learning  time  of  each  system  and 
process. 

•  An  external  organization  should  be  selected  to  maintain  KVA 
databases  for  CCOP  systems.  This  organization  would  act  as 
the  central  repository  for  system  performance  data  to  provide 
reports  and  analysis  on  a  quarterly  or  semi-annual  basis,  thus 
enabling  program  managers  to  make  informed  acquisition 
decisions.  This  data  could  be  expanded  to  include  other  systems 
and  core  processes  to  benchmark  performance  across  the 
enterprise. 


'’Currently  in  process  with  the  Third  Expeditionary  Strike  Group. 
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V 


Conclusion 


The  national  security  environment  of  the  United  States  has  changed 
dramatically  over  the  past  several  years.  In  addition,  sweeping  changes  in  the 
corporate  landscape  have  mandated  changes  in  the  way  government  and 
corporations  manage  fiscal  responsibilities,  investments  and  mitigate  risk.  The 
KVA+RO  framework  is  a  powerful  support  tool  for  decision-makers  in  both  sectors, 
assisting  them  to  meet  those  challenges  more  effectively. 

KVA  is  an  analytic  tool  that  monetizes  the  value,  along  with  cost,  of  each 
process.  It  analyzes  individual  assets  by  asking  how  much  an  asset  contributes  to 
overall  performance  of  the  process.  RO  analysis  estimates  the  total  future  value  of  a 
portfolio  of  options  by  leveraging  uncertainty  and  limiting  downside  risk.  Collectively, 
KVA  and  RO  provide  critical  insights  into  the  performance  of  the  productive  assets  of 
organizations. 
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Appendix  1.  Calculation  Steps  in  KVA  Methodology 


step  1:  Calculate  Time-to-learn. 

•  Time  required  for  average  worker  to  iearn  sub-process  (amount  of  knowiedge  required 
to  produce  a  singie  aggregate  sub-process  output)  is  estimated. 

Step  2:  Calculate  Value  of  Output  (K)  for  each  sub-process. 

•  Caicuiate  amount  of  output  produced  in  sub-process,  by  asset  type  (human  or  if)  by 
counting  the  times  an  asset  executes  a  sub-process. 

•  Muitipiy  that  figure  by  the  time  required  to  iearn  sub-process. 

Step  3:  Calculate  Total  K  for  process. 

•  Once  K  for  each  asset  has  been  caicuiated,  sum  it  to  reveai  totai  K  for  each  sub¬ 
process. 

Steps  4  &  5:  Derive  Market  Comparables  Revenue  surrogate  and  develop  the  Equation 
Numerator  by  assigning  revenue  streams  to  sub-processes. 

•  Under  the  market  comparabies  approach,  muitipiy  totai  number  of  outputs  by  the 
average  market  price-per-unit  to  yieid  a  Surrogate  Revenue. 

Step  6:  Deveiop  the  vaiue  equation  denominator  by  assigning  costs  to  sub-processes. 

•  Assign  costs  directiy  to  each  sub-process  based  on  the  assets  producing  outputs  in 
each. 

•  Divide  cost  of  infrastructure i  assets  assigned  to  muitipie  processes  eveniy  throughout 
those  sub-processes. 

•  Sum  costs  of  assets  in  each  sub-process  to  yieid  the  totai  cost  per  sub-process. 

Step  7;  Caicuiate  productivity,  profitabiiity  and  vaiue  ratios  of  ROi,  ROKA  or  RKOI 

.  Calculate  the  value  equations  of  ROK,  ROKA  and  ROKI  using  revenues  and  costs  assigned  to 
sub-processes,  people  and  IT. 
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Appendix  2.  Current  Limitations  of  KVA 
Methodology 


The  case  study  revealed  a  few  limitations  to  the  implementation  of  KVA  on 
the  ISR  process  that  are  being  addressed.  These  limitations  are; 

.  With  the  raw  data  required  for  the  analysis  residing  in 
multiple  databases  of  varying  classification  levels,  data- 
gathering  mechanisms  that  are  less  human-intensive  and 
more  automated  need  to  be  created  to  extract  the  required 
information. 

.  Although  the  ICP  in  this  case  study  was  developed  through 
the  use  of  subject-matter  experts,  a  standard  description 
and  definition  of  each  sub-process  should  be  reached 
through  an  Intelligence  Community-wide  consensus  of 
process  stake-holders. 

.  A  more  detailed  research  should  be  conducted  to  analyze 
the  knowledge  embedded  in  each  IT  system  to  accurately 
capture  the  benefits  resulting  from  the  execution  of 
particular  system  processes. 

.  The  Market  Comparables  approach  to  valuing  the  outputs  of 
non-profit  organizations,  although  used  as  a  rough  baseline 
to  monetize  outputs  in  this  case  study,  requires  a  more  in- 
depth  look  at  comparable  organizations  utilizing  similar 
processes  to  produce  similar  outputs.  A  broad  database  of 
such  organizations  is  currently  being  created  to  benchmark 
industries  by  functional  groupings  and  products. 

•  To  provide  a  more  powerful  analysis  of  the  ICP,  a  database 
of  comparable  historical  KVA  information  should  be  created 
to  benchmark  future  work  or  to  provide  a  broader  insight  for 
current  work. 
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Appendix  3.  Select  List  of  Organizations  Who  Have 
Applied  KVA  Methodology 


Airtouch 

Arthur  Anderson 

Community  Financial  Resource  Center  (non-profit) 
Courthouse  Athletic  Club 
Direct  TV 

Department  of  Defense  (numerous  applications) 
Earl  Security 

Financial  Services  Consultants  LTD. 

First  Sierra  Financial 

First  World  Communications 

Hughes  Space  and  Communications  Company 

IBM  GLOBAL  SERVICES 

Info.  Technologies 

Internet  Productions 

Lockheed  Martin  Telecommunications 

Morey  Bodyboard  (Mattel  Corp.) 

Mullen 
Nevada  Bell 
NOVA  Chemical 
Pacific  Bell 
Richwood  Industries 
AT  &  T  (SBC,  Inc.) 

SpectraNet  International 
Stentor 

Tokio  Marine  and  Fire 
Toyota  Motors  Sales 
TRW 
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Appendix  4.  Case  Study  for  Possible  Use  in  Classes 
(Derived  from  the  research  study) 


Operational  Case  Study:  Acquiring  and  Deploying  Information 
Technology 

The  proverbial  $64  Billion  dollar  question  is:  “What  investment  in  IT  provides 
the  best  return  and  how  should  I  deploy  the  IT  for  best  effect?”  Answering  this  two 
part  questions  involves  estimating  the  kinds  of  return  on  investment  (ROI)  various  IT 
investments  options  will  provide  as  well  as  how  best  to  deploy  the  IT  to  achieve  the 
desire  effects  on  operations. 

This  case  study  asks  you  to  examine  the  various  ROIs  provided  by  four 
intelligence  gathering  (i.e.,  signal  intelligence)  IT  systems  in  terms  of  how  you  would 
invest  (or  not  invest)  in  each.  It  also  asks  you  to  identify  the  various  options  for 
deploying  the  systems.  At  the  end  of  the  day,  what  really  matters  is  what  the  IT  does 
for  operators  and  their  respective  productivity. 

Problem  Overview 

The  United  States  has  experienced  dramatic  changes  in  national  security 
over  the  past  15  years,  shifting  away  from  the  conventional  threats  posed  during  the 
Cold  War  era  to  more  unconventional  threats  as  evidenced  by  the  tragic  events  of 
September  1 1  and  the  continuing  global  war  on  terrorism.  To  meet  the  challenges 
of  the  new  national  security  environment,  the  Department  of  Defense  (DoD)  plans  to 
spend  $1 .3  trillion  between  2005  and  2009  for  major  programs  ranging  from  new 
intelligence  programs  to  homeland  defense  and  military  operations  oversees.^® 


’°GAO,  “Better  Support  of  Weapon  System  Program  Managers  Needed  to  Improve  Outcomes,”  Nov. 
2005. 
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This  case  study  focuses  on  the  cryptologic  carry-on  program  (CCOP)  portfolio 
of  intelligence  information  systems  and  in  particular,  the  ship  borne  signals 
intelligence  collection  process.  The  USS  Readiness  is  outfitted  to  conduct 
Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  missions  and  has  a  contingent 
of  information  warfare  operators  performing  intelligence  collection  processes  utilizing 
CCOP  systems.  Principal  sub-processes  in  the  ICP  are  shown  in  the  following 
diagram. 


The  Intelligence  Collection  Process 


P3 


The  warship  is  equipped  with  four  CCOP  systems  (A,  B,  C,  and  D).  CCOP 
systems  may  be  used  in  a  single  sub-process  or  across  sub-processes,  and  some 
systems  such  as  CCOP  A  are  highly  complex  with  multiple  subsystems.  Each  sub¬ 
process  is  further  broken  down  into  individual  actions  that  may  be  required  to 
perform  the  sub-process  in  the  intelligence  collection  process.  For  example,  sub¬ 
process  “Target  Data  Processing”  can  be  broken  down  into  a  number  of  human- 
based  tasks  requiring  no  automation. 
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Table  1.  USS  READINESS  CCOP  Systems 


SUB¬ 
PROCESS  NAME 

CCOP 

A 

CCOP 

B 

CCOP 

c 

CCOP 

D 

P1 

Review 

Request/Tasking 

X 

P2 

Determine 

Op/Equip  Mix 

X 

P3 

input  Search 

Function/Coverage  Pian 

X 

P4 

Search/Coiiection 

Process 

X 

X 

P5 

Target  Data 

Acquisition/Capture 

X 

X 

P6 

Target  Data 

Processing 

X 

X 

X 

X 

P7 

Target  Data 

Anaiysis 

X 

X 

X 

P8 

Format  Data  for 

Report  Generation 

X 

P9 

QC  Report 

X 

P10 

Transmit  Report 

X 

ROI  Analysis 

ROI  analysis  was  used  to  compare  two  example  sub-processes:  “Search  and 
Collect”  (P4)  and  “Format  Data  for  Report  Generation”  (P8).  Results  are 
summarized  in  the  following  tables. 
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Table2.  USS  READINESS  Summary  ROI  Results 


Sub-Process 

CCOP  A 

CCOP  B 

CC 

OP  c 

C 

COP  D 

ROI 

Review  Request/Tasking 

1 

68.54% 

22.1 

1% 

Determine  Op/Equip  Mix 

2 

66.86% 

20.8 

9% 

input  Search 

Function/Coverage  Pian 

3 

52.91% 

18.44% 

Search/Coiiection  Process 

4 

830.03% 

48.15% 

239. 

01% 

Target  Data 

Acquisition/Capture 

5 

190.15% 

47.71% 

47.2 

8% 

Target  Data  Processing 

6 

219.39% 

62.59% 

33 

6.13% 

-71.82% 

36.6 

7% 

Target  Data  Anaiysis 

7 

49.98% 

43 

4.76% 

-65.45% 

21.2 

5% 

Format  Data  for  Report 

Generation 

8 

43.34% 

20.37% 

QC  Report 

9 

215.88% 

79.1 

9% 

Transmit  Report 

10 

48.75% 

17.37% 

Metrics  for  Aggregated 

178.56% 

52.81% 

38 

5.44% 

68.63% 

109. 

90% 

•  CCOP  D  is  a  cost-heavy  system  that  executes  very  few  times  with 
negative  ROIs  throughout  the  sample  period,  as  seen  in  Table  2. 
Questions  arising  from  this  analysis  include: 

•  Is  CCOP  D  appropriate  for  this  platform  and  mission? 

•  Are  all  operators  appropriately  trained  in  the  use  of  CCOP  D? 
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•  The  Search  and  Collect  process  (P4)  is  a  high  performer  with  an 
overall  return  of  239%  compared  to  a  -20.37%  return  for  the  Format 
Data  for  Report  Generation  process  (P  8). 

•  What  accounts  for  the  discrepancy  in  the  returns  received  on 
each  process? 

•  What  hurdle  rate  should  be  set  for  each  process? 

•  The  Format  Data  for  Report  Generation  process  (P  8)  only  executes 
once  per  intelligence  report  (process  output)  with  nearly  one  third  of  all 
operators  assigned  to  this  sub-process  (representing  one  fifth  of  the 
total  human  cost  for  the  overall  process). 

•  What  causes  this  low  efficiency  level? 

•  The  Format  Data  for  Report  Generation  process  (P  8)  is  more 
automated  than  P4. 

•  Could  this  process  be  further  automated  or  performed  by  other 
operators  to  yield  higher  efficiency  and  effectiveness  levels? 

Answers  to  these  questions  could  help  program  managers  allocate  funds  to 
new  systems  or  to  existing  systems  for  improve  products  or  to  eliminate  a  system 
from  the  CCOP  portfolio.  Results  could  also  be  used  to  tailor  manning  and  training 
requirements  of  ISR  crews  deploying  CCOP  systems. 


Strategic  Level  Questions 

This  case  was  focused  on  how  the  CCOPs  manager 

With  lives  at  stake  and  billions  of  dollars  at  risk,  difficult  choices  are  made  by 
the  DoD  on  which  projects  to  fund  where  tradeoffs  occur.  Should  investments  be 
made  towards  personnel  or  investments  in  new  technology?  Should  more  funding 
be  allocated  towards  intelligence  collection  or  processing? 
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